Human-Robot Interaction architecture for interactive and lively social robots

Mención Internacional en el título de doctorLa sociedad está experimentando un proceso de envejecimiento que puede provocar un desequilibrio entre la población en edad de trabajar y aquella fuera del mercado de trabajo. Una de las soluciones a este problema que se están considerando hoy en día es la introducción de robots en multiples sectores, incluyendo el de servicios. Sin embargo, para que esto sea una solución viable, estos robots necesitan ser capaces de interactuar con personas de manera satisfactoria, entre otras habilidades. En el contexto de la aplicación de robots sociales al cuidado de mayores, esta tesis busca proporcionar a un robot social las habilidades necesarias para crear interacciones entre humanos y robots que sean naturales. En concreto, esta tesis se centra en tres problemas que deben ser solucionados: (i) el modelado de interacciones entre humanos y robots; (ii) equipar a un robot social con las capacidades expresivas necesarias para una comunicación satisfactoria; y (iii) darle al robot una apariencia vivaz. La solución al problema de modelado de diálogos presentada en esta tesis propone diseñar estos diálogos como una secuencia de elementos atómicos llamados Actos Comunicativos (CAs, por sus siglas en inglés). Se pueden parametrizar en tiempo de ejecución para completar diferentes objetivos comunicativos, y están equipados con mecanismos para manejar algunas de las imprecisiones que pueden aparecer durante interacciones. Estos CAs han sido identificados a partir de la combinación de dos dimensiones: iniciativa (si la tiene el robot o el usuario) e intención (si se pretende obtener o proporcionar información). Estos CAs pueden ser combinados siguiendo una estructura jerárquica para crear estructuras mas complejas que sean reutilizables. Esto simplifica el proceso para crear nuevas interacciones, permitiendo a los desarrolladores centrarse exclusivamente en diseñar el flujo del diálogo, sin tener que preocuparse de reimplementar otras funcionalidades que tienen que estar presentes en todas las interacciones (como el manejo de errores, por ejemplo). La expresividad del robot está basada en el uso de una librería de gestos, o expresiones, multimodales predefinidos, modelados como estructuras similares a máquinas de estados. El módulo que controla la expresividad recibe peticiones para realizar dichas expresiones, planifica su ejecución para evitar cualquier conflicto que pueda aparecer, las carga, y comprueba que su ejecución se complete sin problemas. El sistema es capaz también de generar estas expresiones en tiempo de ejecución a partir de una lista de acciones unimodales (como decir una frase, o mover una articulación). Una de las características más importantes de la arquitectura de expresividad propuesta es la integración de una serie de métodos de modulación que pueden ser usados para modificar los gestos del robot en tiempo de ejecución. Esto permite al robot adaptar estas expresiones en base a circunstancias particulares (aumentando al mismo tiempo la variabilidad de la expresividad del robot), y usar un número limitado de gestos para mostrar diferentes estados internos (como el estado emocional). Teniendo en cuenta que ser reconocido como un ser vivo es un requisito para poder participar en interacciones sociales, que un robot social muestre una apariencia de vivacidad es un factor clave en interacciones entre humanos y robots. Para ello, esta tesis propone dos soluciones. El primer método genera acciones a través de las diferentes interfaces del robot a intervalos. La frecuencia e intensidad de estas acciones están definidas en base a una señal que representa el pulso del robot. Dicha señal puede adaptarse al contexto de la interacción o al estado interno del robot. El segundo método enriquece las interacciones verbales entre el robot y el usuario prediciendo los gestos no verbales más apropiados en base al contenido del diálogo y a la intención comunicativa del robot. Un modelo basado en aprendizaje automático recibe la transcripción del mensaje verbal del robot, predice los gestos que deberían acompañarlo, y los sincroniza para que cada gesto empiece en el momento preciso. Este modelo se ha desarrollado usando una combinación de un encoder diseñado con una red neuronal Long-Short Term Memory, y un Conditional Random Field para predecir la secuencia de gestos que deben acompañar a la frase del robot. Todos los elementos presentados conforman el núcleo de una arquitectura de interacción humano-robot modular que ha sido integrada en múltiples plataformas, y probada bajo diferentes condiciones. El objetivo central de esta tesis es contribuir al área de interacción humano-robot con una nueva solución que es modular e independiente de la plataforma robótica, y que se centra en proporcionar a los desarrolladores las herramientas necesarias para desarrollar aplicaciones que requieran interacciones con personas.Society is experiencing a series of demographic changes that can result in an unbalance between the active working and non-working age populations. One of the solutions considered to mitigate this problem is the inclusion of robots in multiple sectors, including the service sector. But for this to be a viable solution, among other features, robots need to be able to interact with humans successfully. This thesis seeks to endow a social robot with the abilities required for a natural human-robot interactions. The main objective is to contribute to the body of knowledge on the area of Human-Robot Interaction with a new, platform-independent, modular approach that focuses on giving roboticists the tools required to develop applications that involve interactions with humans. In particular, this thesis focuses on three problems that need to be addressed: (i) modelling interactions between a robot and an user; (ii) endow the robot with the expressive capabilities required for a successful communication; and (iii) endow the robot with a lively appearance. The approach to dialogue modelling presented in this thesis proposes to model dialogues as a sequence of atomic interaction units, called Communicative Acts, or CAs. They can be parametrized in runtime to achieve different communicative goals, and are endowed with mechanisms oriented to solve some of the uncertainties related to interaction. Two dimensions have been used to identify the required CAs: initiative (the robot or the user), and intention (either retrieve information or to convey it). These basic CAs can be combined in a hierarchical manner to create more re-usable complex structures. This approach simplifies the creation of new interactions, by allowing developers to focus exclusively on designing the flow of the dialogue, without having to re-implement functionalities that are common to all dialogues (like error handling, for example). The expressiveness of the robot is based on the use of a library of predefined multimodal gestures, or expressions, modelled as state machines. The module managing the expressiveness receives requests for performing gestures, schedules their execution in order to avoid any possible conflict that might arise, loads them, and ensures that their execution goes without problems. The proposed approach is also able to generate expressions in runtime based on a list of unimodal actions (an utterance, the motion of a limb, etc...). One of the key features of the proposed expressiveness management approach is the integration of a series of modulation techniques that can be used to modify the robot’s expressions in runtime. This would allow the robot to adapt them to the particularities of a given situation (which would also increase the variability of the robot expressiveness), and to display different internal states with the same expressions. Considering that being recognized as a living being is a requirement for engaging in social encounters, the perception of a social robot as a living entity is a key requirement to foster human-robot interactions. In this dissertation, two approaches have been proposed. The first method generates actions for the different interfaces of the robot at certain intervals. The frequency and intensity of these actions are defined by a signal that represents the pulse of the robot, which can be adapted to the context of the interaction or the internal state of the robot. The second method enhances the robot’s utterance by predicting the appropriate non-verbal expressions that should accompany them, according to the content of the robot’s message, as well as its communicative intention. A deep learning model receives the transcription of the robot’s utterances, predicts which expressions should accompany it, and synchronizes them, so each gesture selected starts at the appropriate time. The model has been developed using a combination of a Long-Short Term Memory network-based encoder and a Conditional Random Field for generating a sequence of gestures that are combined with the robot’s utterance. All the elements presented above conform the core of a modular Human-Robot Interaction architecture that has been integrated in multiple platforms, and tested under different conditions.Programa de Doctorado en Ingeniería Eléctrica, Electrónica y Automática por la Universidad Carlos III de MadridPresidente: Fernando Torres Medina.- Secretario: Concepción Alicia Monje Micharet.- Vocal: Amirabdollahian Farshi

Localización de robots en mapas 3D mediante CLONALG

El objetivo de este Trabajo Fin de Grado es aplicar el Algoritmo de Selección Clonal (comúnmente conocido como CLONALG) al filtro de localización global de alta precisión para entornos 3D, también llamado RELF-3D, desarrollado por Fernando Martín, Luis Moreno, Santiago Garrido y Dolores Blanco, investigadores de la Universidad Carlos III de Madrid. Una vez se desarrolle el algoritmo genético y se integre en el programa completo ya existente, se pasará a realizar pruebas para poder comparar los resultados obtenidos con el CLONALG y los obtenidos mediante el algoritmo de Evolución Diferencial, y así poder tener una base para decidir cuál de los dos métodos es más eficaz a la hora de cumplir con su propósito. En esta memoria, se comenzará con una introducción al trabajo, en la que se expondrán los objetivos del proyecto, se verá el trabajo previo y el estado del arte. En el siguiente apartado, se le proporcionará al lector una pequeña base en el campo de los algoritmos genéticos, para posteriormente explicar con mayor profundidad el método de Evolución Diferencial, comenzando por las bases teóricas y acabando con un estudio acerca del algoritmo implementado en el programa. A continuación, se llega a la parte central de este trabajo, el CLONALG. Se entrará en profundidad en el funcionamiento del algoritmo, tanto teórica como prácticamente, además de explicar las diferentes partes del código desarrollado. En este apartado también se expondrá el proceso de desarrollo de la solución, las decisiones de diseño y los problemas a los que fue necesario hacer frente para alcanzar la solución. Por último, se compararán los resultados obtenidos por ambos métodos, además de realizar un análisis de sensibilidad del CLONALG con respecto a una serie de parámetros, y se cerrará la memoria con una conclusión acerca de los resultados obtenidos en la investigación.The objective of this Final Project is to apply the Clonal Selection Algorithm (commonly known as CLONALG) to the Rejection Evolutionary Localization Filter in 3D, also called RELF-3D, developed by Fernando Martín, Luis Moreno, Santiago Garrido and Dolores White, researchers at the Carlos III University of Madrid. Once the genetic algorithm is developed and integrated into the existing full program, tests will be performed in order to compare the results obtained with CLONALG and those obtained by the algorithm Differential Evolution, so we can have a basis for deciding which of the two methods is more effective in fulfilling its purpose. In this report, we will begin with an introduction to the work, in which the project objectives will be discussed, previous work and the state of the art will be seen. In the next section, we will provide the reader a small introduction into the field of genetic algorithms to further explain in detail the method of Differential Evolution, beginning with the theoretical bases and ending with a study of the algorithm implemented in the program. Then we will reach the central part of this work, the CLONALG. It will go into depth on the performance of the algorithm, both theoretically and practically, in addition to explaining the different parts of the code developed. This section describes the development process of the solution and also expose the design decisions and the problems it was necessary to solve to reach the solution. Finally, the results obtained by both methods, in addition to an analysis of CLONALG sensitivity with respect to a number of parameters are compared, and the report will end with a conclusion about the results of the investigation.Ingeniería en Tecnologías Industriale

Do you want to make your robot warmer? Make it more reactive!

Endowing robots with the ability to respond appropriately to stimuli contributes to the perception of an illusion of "life" in robots, which is determinant for their acceptance as companions. This work aims to study how a series of bio-inspired reactive responses impact on the way in which participants perceive a social robot. In particular, the proposed system endows the robot with the ability to react to stimuli that are not only related to the current task but are also related to other external events. We conducted an experiment where the participants observed a video-recorded interaction with two robots: one was able to respond to both task-related and non-task-related events, while the other was only able to react to task-related events. To evaluate the experiment, we used the RoSAS questionnaire. The results yielded significant differences for two factors, showing that the addition of responses to non-task-related stimuli increased the robot¿s warmth and competence.This work was supported in part by the Robots Sociales para Estimulación Física, Cognitiva y Afectiva de Mayores (RoSEs) funded by the Ministerio de Ciencia, Innovación y Universidades, Spanish Government under Grant RTI2018-096338-B-I00; in part by the Robots sociales para mitigar la soledad y el aislamiento en mayores (SoRoLI) funded by Agencia Estatal de Investigación, Ministerio de Ciencia e Innovación, Spanish Government under Grant PID2021-123941OA-I00; and in part by the Multiannual Agreement with UC3M (“Fostering Young Doctors Research”) in the context of the V PRICIT (Research and Technological Innovation Regional Programme) by the Madrid Government (Comunidad de Madrid-Spain) under Grant SMM4HRI-CM-UC3M

Modelling Multimodal Dialogues for Social Robots Using Communicative Acts

Social Robots need to communicate in a way that feels natural to humans if they are to effectively bond with the users and provide an engaging interaction. Inline with this natural, effective communication, robots need to perceive and manage multimodal information, both as input and output, and respond accordingly. Consequently, dialogue design is a key factor in creating an engaging multimodal interaction. These dialogues need to be flexible enough to adapt to unforeseen circumstances that arise during the conversation but should also be easy to create, so the development of new applications gets simpler. In this work, we present our approach to dialogue modelling based on basic atomic interaction units called Communicative Acts. They manage basic interactions considering who has the initiative (the robot or the user), and what is his/her intention. The two possible intentions are either ask for information or give information. In addition, because we focus on one-to-one interactions, the initiative can only be taken by the robot or the user. Communicative Acts can be parametrised and combined in a hierarchical manner to fulfil the needs of the robot’s applications, and they have been equipped with built-in functionalities that are in charge of low-level communication tasks. These tasks include communication error handling, turn-taking or user disengagement. This system has been integrated in Mini, a social robot that has been created to assist older adults with cognitive impairment. In a case of use, we demonstrate the operation of our system as well as its performance in real human–robot interactions.The research leading to these results has received funding from the projects Development of social robots to help seniors with cognitive impairment (ROBSEN), funded by the Ministerio de Economia y Competitividad; RoboCity2030-DIH-CM, Madrid Robotics Digital Innovation Hub, S2018/NMT-4331, funded by “Programas de Actividades I+D en la Comunidad de Madrid” and cofunded by Structural Funds of the EU; and Robots sociales para estimulación física, cognitiva y afectiva de mayores (ROSES) RTI2018-096338-B-I00 funded by Agencia Estatal de Investigación (AEI), Ministerio de Ciencia, Innovación y Universidade

Emotion and mood blending in embodied artificial agents: expressing affective states in the mini social robot

Robots that are devised for assisting and interacting with humans are becoming fundamental in many applications, including in healthcare, education, and entertainment. For these robots, the capacity to exhibit affective states plays a crucial role in creating emotional bonding with the user. In this work, we present an affective architecture that grounds biological foundations to shape the affective state of the Mini social robot in terms of mood and emotion blending. The affective state depends upon the perception of stimuli in the environment, which influence how the robot behaves and affectively communicates with other peers. According to research in neuroscience, mood typically rules our affective state in the long run, while emotions do it in the short term, although both processes can overlap. Consequently, the model that is presented in this manuscript deals with emotion and mood blending towards expressing the robot's internal state to the users. Thus, the primary novelty of our affective model is the expression of: (i) mood, (ii) punctual emotional reactions to stimuli, and (iii) the decay that mood and emotion undergo with time. The system evaluation explored whether users can correctly perceive the mood and emotions that the robot is expressing. In an online survey, users evaluated the robot's expressions showing different moods and emotions. The results reveal that users could correctly perceive the robot's mood and emotion. However, emotions were more easily recognized, probably because they are more intense affective states and mainly arise as a stimuli reaction. To conclude the manuscript, a case study shows how our model modulates Mini's expressiveness depending on its affective state during a human-robot interaction scenario.The research leading to these results has received funding from the projects Robots sociales para estimulación física, cognitiva y afectiva de mayores (ROSES) RTI2018-096338-B-I00 funded by Agencia Estatal de Investigación (AEI), Ministerio de Ciencia, Innovación y Universidades and RoboCity2030-DIH-CM, Madrid Robotics Digital Innovation Hub, S2018/NMT-4331, funded by "Programas de Actividades I+D en la Comunidad de Madrid" and cofunded by Structural Funds of the EU. Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature

Non-verbal gesture prediction using deep learning

[Resumen] En años recientes, la robótica está empezando a usarse fuera de aplicaciones industriales, y los robots empiezan ya a tomar parte en tareas que requieren interactuar con personas. Para que estas interacciones resulten naturales, es necesario que el robot sea capaz de ejecutar expresiones de forma autónoma. En situaciones donde el robot está hablando, los gestos no verbales que ejecute deben apoyar el mensaje comunicativo de la componente verbal, y ambas componentes deben estar sincronizadas apropiadamente. En este trabajo presentamos un sistema de predicción de gestos no verbales para robots sociales basado en uno de los avances más significativos en años recientes en el campo del aprendizaje profundo: el modelo transformer. Esta solución será comparada con un modelo previo que combina redes recurrentes con campos aleatorios condicionales para resolver la misma tarea. Los resultados de la comparación de ambos modelos indican que, al igual que en otras tareas de procesamiento del lenguaje natural, los transformers presentan una clara mejora a la hora de resolver la tarea de predecir gestos no verbales para robots sociales.[Abstract] In recent years, robotics is starting to expand beyond industrial applications, and robots are starting to take part in tasks that require interacting with human beings. For this interactions to be natural for the users, it is necessary that the robots are capable of performing expressions autonomously. In situations where the robot is speaking, the non-verbal gestures performed by the robot must also support the communicative message expressed by the verbal component, and both components should be properly synchronized. In this work, we present a gesture prediction system for social robots based in one of the most significant advances in the area of deep learning: the transformer model. This solution will be compared with a previous system based on a combination of recurrent neural networks and conditional random fields. The results of the comparison conducted show that, as it is the case for other tasks in the field of natural language processing, transformers present a clear improvement for the task of predicting non-verbal expressions for social robots.Ministerio de Ciencia e Innovación; TED2021-132079B-I00Comunidad de Madrid; SMM4HRI-CM-UC3

Projection Surfaces Detection and Image Correction for Mobile Robots in HRI

Projectors have become a widespread tool to share information in Human-Robot Interaction with large groups of people in a comfortable way. Finding a suitable vertical surface becomes a problem when the projector changes positions when a mobile robot is looking for suitable surfaces to project. Two problems must be addressed to achieve a correct undistorted image: (i) finding the biggest suitable surface free from obstacles and (ii) adapting the output image to correct the distortion due to the angle between the robot and a nonorthogonal surface. We propose a RANSAC-based method that detects a vertical plane inside a point cloud. Then, inside this plane, we apply a rectangle-fitting algorithm over the region in which the projector can work. Finally, the algorithm checks the surface looking for imperfections and occlusions and transforms the original image using a homography matrix to display it over the area detected. The proposed solution can detect projection areas in real-time using a single Kinect camera, which makes it suitable for applications where a robot interacts with other people in unknown environments. Our Projection Surfaces Detector and the Image Correction module allow a mobile robot to find the right surface and display images without deformation, improving its ability to interact with people

Mini: A New Social Robot for the Elderly

The unceasing aging of the population is leading to new problems in developed countries. Robots represent an opportunity to extend the period of independent living of the elderly as well as to ameliorate their economic burden and social problems. We present a new social robot, Mini, specifically designed to assist and accompany the elderly in their daily life either at home or in a nursing facility. Based on the results of several meetings with experts in this field, we have built a robot able to provide services in the areas of safety, entertainment, personal assistance and stimulation. Mini supports elders and caregivers in cognitive and mental tasks. We present the robot platform and describe the software architecture, particularly focussing on the human–robot interaction. We give in detail how the robot operates and the interrelation of the different modules of the robot in a real use case. In the last part of the paper, we evaluated how users perceive the robot. Participants reported interesting results in terms of usability, appearance, and satisfaction. This paper describes all aspects of the design and development of a new social robot that can be used by other researchers who face the multiple challenges of creating a new robotic platform for older people.The research leading to these results has received funding from the projects: Development of social robots to help seniors with cognitive impairment (ROBSEN), funded by the Ministerio de Economía y Competitividad; and Robots Sociales para Estimulación Física, Cognitiva y Afectiva de Mayores (ROSES), funded by the Ministerio de Ciencia, Innovación y Universidades.Publicad

Surgical site infection after gastrointestinal surgery in children : an international, multicentre, prospective cohort study

Introduction Surgical site infection (SSI) is one of the most common healthcare-associated infections (HAIs). However, there is a lack of data available about SSI in children worldwide, especially from low-income and middle-income countries. This study aimed to estimate the incidence of SSI in children and associations between SSI and morbidity across human development settings. Methods A multicentre, international, prospective, validated cohort study of children aged under 16 years undergoing clean-contaminated, contaminated or dirty gastrointestinal surgery. Any hospital in the world providing paediatric surgery was eligible to contribute data between January and July 2016. The primary outcome was the incidence of SSI by 30 days. Relationships between explanatory variables and SSI were examined using multilevel logistic regression. Countries were stratified into high development, middle development and low development groups using the United Nations Human Development Index (HDI). Results Of 1159 children across 181 hospitals in 51 countries, 523 (45 center dot 1%) children were from high HDI, 397 (34 center dot 2%) from middle HDI and 239 (20 center dot 6%) from low HDI countries. The 30-day SSI rate was 6.3% (33/523) in high HDI, 12 center dot 8% (51/397) in middle HDI and 24 center dot 7% (59/239) in low HDI countries. SSI was associated with higher incidence of 30-day mortality, intervention, organ-space infection and other HAIs, with the highest rates seen in low HDI countries. Median length of stay in patients who had an SSI was longer (7.0 days), compared with 3.0 days in patients who did not have an SSI. Use of laparoscopy was associated with significantly lower SSI rates, even after accounting for HDI. Conclusion The odds of SSI in children is nearly four times greater in low HDI compared with high HDI countries. Policies to reduce SSI should be prioritised as part of the wider global agenda.Peer reviewe

Jornadas Nacionales de Robótica y Bioingeniería 2023: Libro de actas

Las Jornadas de Robótica y Bioingeniería de 2023 tienen lugar en la Escuela Técnica Superior de Ingeniería Industrial de la Universidad Politécnica de IVIadrid, entre los días 14 y 16 de junio de 2023. En este evento propiciado por el Comité Español de Automática (CEA) tiene lugar la celebración conjunta de las XII Jornadas Nacionales de Robótica y el XIV Simposio CEA de Bioingeniería. Las Jornadas Nacionales de Robótica es un evento promovido por el Grupo Temático de Robótica (GTRob) de CEA para dar visibilidad y mostrar las actividades desarrolladas en el ámbito de la investigación y transferencia tecnológica en robótica. Asimismo, el propósito de Simposio de Bioingeniería, que cumple ahora su decimocuarta dicción, es el de proporcionar un espacio de encuentro entre investigadores, desabolladores, personal clínico, alumnos, industriales, profesionales en general e incluso usuarios que realicen su actividad en el ámbito de la bioingeniería. Estos eventos se han celebrado de forma conjunta en la anualidad 2023. Esto ha permitido aunar y congregar un elevado número de participantes tanto de la temática robótica como de bioingeniería (investigadores, profesores, desabolladores y profesionales en general), que ha posibilitado establecer puntos de encuentro, sinergias y colaboraciones entre ambos. El programa de las jornadas aúna comunicaciones científicas de los últimos resultados de investigación obtenidos, por los grupos a nivel español más representativos dentro de la temática de robótica y bioingeniería, así como mesas redondas y conferencias en las que se debatirán los temas de mayor interés en la actualidad. En relación con las comunicaciones científicas presentadas al evento, se ha recibido un total de 46 ponencias, lo que sin duda alguna refleja el alto interés de la comunidad científica en las Jornadas de Robótica y Bioingeniería. Estos trabajos serán expuestos y presentados a lo largo de un total de 10 sesiones, distribuidas durante los diferentes días de las Jornadas. Las temáticas de los trabajos cubren los principales retos científicos relacionados con la robótica y la bioingeniería: robótica aérea, submarina, terrestre, percepción del entorno, manipulación, robótica social, robótica médica, teleoperación, procesamiento de señales biológicos, neurorehabilitación etc. Confiamos, y estamos seguros de ello, que el desarrollo de las jornadas sea completamente productivo no solo para los participantes en las Jornadas que podrán establecer nuevos lazos y relaciones fructíferas entre los diferentes grupos, sino también aquellos investigadores que no hayan podido asistir. Este documento que integra y recoge todas las comunicaciones científicas permitirá un análisis más detallado de cada una de las mismas